Lecture 4

Estimating Sampling Distributions using the Bootstrap

Point and Interval Estimation

  • Last week, we talked about estimating population proportions.
  • Our point estimates were proportions of a random sample, drawn with replacement, from the population.
  • To we’d be, we used interval estimates. In particular, calibrated interval estimates. Confidence intervals.
    • To get these intervals, we add ‘arms’ of equal length to our point estimate.
    • We choose the arm length so that, 95% of the time, the population proportion is within them.

Calibration of Interval Estimates

  • We can do that using the sampling distribution of our point estimate.
  • We’d draw arms out from the sampling distribution’s until they span 95% of point estimates.
  • We can see that this is the right length by a shift of perspective.
    • A point estimate can touch the mean with its arms iff1 the mean can touch it with equally long arms.
    • So 95% of intervals cover if and only if 95% of point estimates are within the mean’s arms.
  • In practice, we can’t do exactly that. We don’t know the sampling distribution.
  • But we can use an estimate of the sampling distribution in its place.
  • If it’s a good one, we’ll get approximately 95% coverage.

A Parametric Estimate of the Sampling Distribution

  • It helps to know the parametric form of the sampling distribution: it’s binomial.
    • It’s the distribution of the proportion heads in 625 flips of a coin with probability \(p \approx 0.70\) of heads.
    • where \(p \approx 0.70\) is the proportion of registered voters who will vote.
  • By parametric form, I mean a formula in terms of some parameters.
    • That tells us what parameters we need to estimate to estimate the sampling distribution.
    • In this case, the only (unknown) parameter is the probability of heads, \(p\).

A Parametric Estimate of the Sampling Distribution

  • In R, dbinom(x, n, p), calculates this formula where …
    • x is the number of heads
    • n is the number of flips
    • p is the probability of heads, i.e., the proportion \(\bar y\) of registered voters who will vote.
  • To estimate the sampling distribution, we plug in our point estimate \(\bar Y\) for \(p\).
  • There’s another way to think about this that generalizes to more complicated situations.

The Bootstrap Interpretation

  • The estimated sampling distribution is \(\text{Binomial}(n,p)/n\) for \(n=625\) and \(p=\bar Y\approx 0.68\).
    • It’s the distribution of the proportion heads in 625 flips of a coin with probability \(p \approx 0.68\) of heads.
    • where \(p \approx 0.68\) is the proportion of voters we’ve polled who will vote.
  • That is, it’s the sampling distribution of a ‘poll’ of the people in our sample, i.e.
    • roll a 625-sided die 625 times
    • call up the corresponding person in our sample
    • and counting up the yeses we hear
  • Note that this is random because we’re drawing with replacement.
    • Each time we run this poll, we call each person in our sample 0,1,2,… times
    • And the number of times we call them is random.

The Bootstrap Interpretation

  • If we plot our voters on a map, you can see the idea in visual terms.
    • On the left, we have the population.
    • In the middle, we have our sample. It’s drawn, with replacement, from the population.
    • On the right, we have something else. A new sample drawn, with replacement, from the sample.
  • Each ‘call’ that a person receives increases the size of their dot: \(\text{circle area} \propto \text{number of calls}\).
  • In the sample, even though it’s drawn with replacement, all dots are the same size.
    • Because we draw from such a large population, nobody gets called twice.
  • In the bootstrap sample, dots vary in size.
    • Because we draw \(n\) people from a sample of size \(n\), it’s almost impossible not to call somebody twice.

Bootstrapping

  • Before the election, we don’t observe the population. But we do observe the sample.
    • And we can sample from the sample act as if it were the population.
    • We’ll take repeated random samples of size 625, with replacement, from our sample of size 625.
  • We call these bootstrap samples and estimates based on them bootstrap estimates.
    • The distribution of these estimates is called the bootstrap sampling distribution.
    • If the sample is like the population, this should be like our estimator’s actual sampling distribution.

The Sample \[ \begin{array}{r|rrrr|r} i & 1 & 2 & \dots & 625 & \bar{Y}_{625} \\ Y_i & 1 & 1 & \dots & 1 & 0.68 \\ \end{array} \]

The Bootstrap Sample

\[ \begin{array}{r|rrrr|r} i & 1 & 2 & \dots & 625 & \bar{Y}_{625}^* \\ Y_i^* & 1 & 1 & \dots & 1 & 0.63 \\ \end{array} \]

The Population

\[ \begin{array}{r|rrrr|r} j & 1 & 2 & \dots & 7.23M & \bar{y}_{7.23M} \\ y_{j} & 1 & 1 & \dots & 1 & 0.70 \\ \end{array} \]

The ‘Bootstrap Population’ — The Sample \[ \begin{array}{r|rrrr|r} j & 1 & 2 & \dots & 625 & \bar{y}^*_{625} \\ y_j^* & 1 & 1 & \dots & 1 & 0.68 \\ \end{array} \]

The Bootstrap is Nonparametric

bootstrap.samples = array(dim=10000)
for(rr in 1:10000) {
    Y.star = Y[sample(1:n, n, replace=TRUE)]
    bootstrap.samples[rr] = mean(Y.star) 
}

  • We do not need to know the parametric form of our sampling distribution to use it.
  • All we do is re-run our poll acting as if our sample were the population.
    • We can do this no matter what we’re estimating.
    • So let’s try it out on some other stuff.

Comparing Black and Non-Black Turnout

\[ \small{ \begin{array}{r|rr|rr|r|rr|rrrrr} \text{call} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{question} & X_1 & Y_1 & X_2 & Y_2 & \dots & X_{625} & Y_{625} & \overline{X}_{625} & \overline{Y}_{625} &\frac{\sum_{i:X_i=0} Y_i}{\sum_{i:X_i=0} 1} & \frac{\sum_{i:X_i=1} Y_i}{\sum_{i:X_i=1} 1} & \text{difference} \\ \text{outcome} & \underset{\textcolor{gray}{x_{869369}}}{0} & \underset{\textcolor{gray}{y_{869369}}}{1} & \underset{\textcolor{gray}{x_{4428455}}}{1} & \underset{\textcolor{gray}{y_{4428455}}}{1} & \dots & \underset{\textcolor{gray}{x_{1268868}}}{0} & \underset{\textcolor{gray}{y_{1268868}}}{1} & 0.28 & 0.68 & 0.68 & 0.69 & 0.01 \\ \end{array} } \]

  • It’s useful to predict turnout among Black voters because they tend to vote differently than non-Black voters.
  • Let’s suppose that we’re interested in the difference in turnout between Black and non-Black voters.
  • This is a bit reductive, but it’s the beginning of the semester, so we’re keeping things simple.
  • We know from the voter file that 30% of registered voters are Black.
  • Let’s suppose we asked the people we polled if they were Black, recording their answer in a covariate \(X_i\).

Comparing Black and Non-Black Turnout

\[ \small{ \begin{array}{r|rr|rr|r|rr|rrrrr} \text{call} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{question} & X_1 & Y_1 & X_2 & Y_2 & \dots & X_{625} & Y_{625} & \overline{X}_{625} & \overline{Y}_{625} &\frac{\sum_{i:X_i=0} Y_i}{\sum_{i:X_i=0} 1} & \frac{\sum_{i:X_i=1} Y_i}{\sum_{i:X_i=1} 1} & \text{difference} \\ \text{outcome} & \underset{\textcolor{gray}{x_{869369}}}{0} & \underset{\textcolor{gray}{y_{869369}}}{1} & \underset{\textcolor{gray}{x_{4428455}}}{1} & \underset{\textcolor{gray}{y_{4428455}}}{1} & \dots & \underset{\textcolor{gray}{x_{1268868}}}{0} & \underset{\textcolor{gray}{y_{1268868}}}{1} & 0.28 & 0.68 & 0.68 & 0.69 & 0.01 \\ \end{array} } \]

  • In our poll, we found that…
    • 172 of the people we called (28%) were Black with a turnout rate of 0.69
    • 453 of the people we called (72%) were non-Black with a turnout rate of 0.68
  • After the election, we found that …
    • Turnout among Black registered voters was 0.74
    • Turnout among non-Black registered voters was 0.68.
  • We’d estimate the difference to be \(0.69-0.68 \approx 0.01\).
  • The actual difference was \(0.74-0.68 \approx 0.06\).
  • Depending on what you’re doing, that point estimate might be good enough or might not be.
  • It’d be nice to have an interval estimate.

A Table of Imaginary Polls

$$ \[\begin{array}{r|rr|rr|r|rr|rrrr} \text{call} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{poll} & X_1 & Y_1 & X_2 & Y_2 & \dots & X_{625} & Y_{625} & \overline{X} & \overline{Y} &\frac{\sum_{i:X_i=0} Y_i}{\sum_{i:X_i=0} 1} & \frac{\sum_{i:X_i=1} Y_i}{\sum_{i:X_i=1} 1} & \text{difference} \\ \hline \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}\dots & \color[RGB]{7,59,76}0 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0.28 & \color[RGB]{7,59,76}0.68 & \color[RGB]{7,59,76}0.68 & \color[RGB]{7,59,76}0.69 & \color[RGB]{7,59,76}0.01 \\ \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}0 & \color[RGB]{239,71,111}0 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}\dots & \color[RGB]{239,71,111}0 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}0.29 & \color[RGB]{239,71,111}0.71 & \color[RGB]{239,71,111}0.72 & \color[RGB]{239,71,111}0.70 & \color[RGB]{239,71,111}-0.01 \\ \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}0 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}\dots & \color[RGB]{17,138,178}0 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}0.26 & \color[RGB]{17,138,178}0.70 & \color[RGB]{17,138,178}0.68 & \color[RGB]{17,138,178}0.76 & \color[RGB]{17,138,178}0.08 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}\dots & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}0.30 & \color[RGB]{6,214,160}0.71 & \color[RGB]{6,214,160}0.68 & \color[RGB]{6,214,160}0.79 & \color[RGB]{6,214,160}0.12 \\ \end{array}\]

$$

The Sampling Distribution

difference.samples = array(dim=10000)
for(rr in 1:10000) {
    I = sample(1:m, n, replace=TRUE)
    X = x[I]
    Y = y[I]
    difference.samples[rr] = mean(Y[X==1]) - mean(Y[X==0])
}

  • Here’s what the sampling distribution of this difference in turnout looks like.
  • As before, if we can estimate it we can use that estimate to get a 95% confidence interval.
  • But, unlike before, we don’t really know the parametric form of its sampling distribution.
  • Or—at least—it’d be a pain to work it out. So we’ll use the bootstrap to estimate it.

A Table of Bootstrap Polls

\[ \small{ \begin{array}{r|rr|rr|r|rr|rrrr} \text{call} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{poll} & X_1 & Y_1 & X_2 & Y_2 & \dots & X_{625} & Y_{625} & \overline{X} & \overline{Y} &\frac{\sum_{i:X_i=0} Y_i}{\sum_{i:X_i=0} 1} & \frac{\sum_{i:X_i=1} Y_i}{\sum_{i:X_i=1} 1} & \text{difference} \\ \hline \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}\dots & \color[RGB]{7,59,76}0 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0.28 & \color[RGB]{7,59,76}0.68 & \color[RGB]{7,59,76}0.68 & 0.69 & \color[RGB]{7,59,76}0.01 \end{array} } \]

\[ \small{ \begin{array}{r|rr|rr|r|rr|rrrr} \text{`call'} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{`poll'} & X_1^* & Y_1^* & X_2^* & Y_2^* & \dots & X^*_{625} & Y^*_{625} & \overline{X}^* & \overline{Y}^* &\frac{\sum_{i:X_i^*=0} Y_i}{\sum_{i:X_i^*=0} 1} & \frac{\sum_{i:X_i^*=1} Y_i}{\sum_{i:X_i^*=1} 1} & \text{difference} \\ \hline \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}X_{398} & \color[RGB]{239,71,111}Y_{398} & & & \color[RGB]{239,71,111}\dots & & & & & & & & \\ & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}0 & & & \color[RGB]{239,71,111}\dots & & & & & & & & \\ \color[RGB]{17,138,178}2 & \color[RGB]{17,138,178}X_{293} & \color[RGB]{17,138,178}Y_{293} & & & \color[RGB]{17,138,178}\dots & & & & & & & & \\ & \color[RGB]{17,138,178}0 & \color[RGB]{17,138,178}1 & & & \color[RGB]{17,138,178}\dots & & & & & & & & \\ \color[RGB]{6,214,160}3 & \color[RGB]{6,214,160}X_{281} & \color[RGB]{6,214,160}Y_{281} & & & \color[RGB]{6,214,160}\dots & & & & & & & & \\ & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}0 & & & \color[RGB]{6,214,160}\dots & & & & & & & & \\ \end{array} } \]

A Table of Bootstrap Polls

\[ \small{ \begin{array}{r|rr|rr|r|rr|rrrr} \text{call} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{poll} & X_1 & Y_1 & X_2 & Y_2 & \dots & X_{625} & Y_{625} & \overline{X} & \overline{Y} &\frac{\sum_{i:X_i=0} Y_i}{\sum_{i:X_i=0} 1} & \frac{\sum_{i:X_i=1} Y_i}{\sum_{i:X_i=1} 0} & \text{difference} \\ \hline \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}\dots & \color[RGB]{7,59,76}0 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0.28 & \color[RGB]{7,59,76}0.68 & \color[RGB]{7,59,76}0.68 & 0.69 & \color[RGB]{7,59,76}0.01 \end{array} } \]

\[ \small{ \begin{array}{r|rr|rr|r|rr|rrrr} \text{`call'} & 1 & & 2 & & \dots & 625 & & & & & & \\ \text{`poll'} & X_1^* & Y_1^* & X_2^* & Y_2^* & \dots & X^*_{625} & Y^*_{625} & \overline{X}^* & \overline{Y}^* &\frac{\sum_{i:X_i^*=0} Y_i}{\sum_{i:X_i^*=0} 1} & \frac{\sum_{i:X_i^*=1} Y_i}{\sum_{i:X_i^*=1} 1} & \text{difference} \\ \hline \color[RGB]{239,71,111}2 & \color[RGB]{239,71,111}X_{398} & \color[RGB]{239,71,111}Y_{398} & \color[RGB]{239,71,111}X_{129} & \color[RGB]{239,71,111}Y_{129} & \color[RGB]{239,71,111}\dots & \color[RGB]{239,71,111}X_{232} & \color[RGB]{239,71,111}Y_{232} & & & & & & \\ & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}0 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}\dots & \color[RGB]{239,71,111}0 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}0.29 & \color[RGB]{239,71,111}0.68 & \color[RGB]{239,71,111}0.69 & \color[RGB]{239,71,111}0.68 & \color[RGB]{239,71,111}-0.01 \\ \color[RGB]{17,138,178}2 & \color[RGB]{17,138,178}X_{293} & \color[RGB]{17,138,178}Y_{293} & \color[RGB]{17,138,178}X_{526} & \color[RGB]{17,138,178}Y_{526} & \color[RGB]{17,138,178}\dots & \color[RGB]{17,138,178}X_{578} & \color[RGB]{17,138,178}Y_{578} & & & & & & \\ & \color[RGB]{17,138,178}0 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}0 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}\dots & \color[RGB]{17,138,178}0 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}0.28 & \color[RGB]{17,138,178}0.65 & \color[RGB]{17,138,178}0.64 & \color[RGB]{17,138,178}0.67 & \color[RGB]{17,138,178}0.03 \\ \color[RGB]{6,214,160}1M & \color[RGB]{6,214,160}X_{281} & \color[RGB]{6,214,160}Y_{281} & \color[RGB]{6,214,160}X_{520} & \color[RGB]{6,214,160}Y_{520} & \color[RGB]{6,214,160}\dots & \color[RGB]{6,214,160}X_{363} & \color[RGB]{6,214,160}Y_{363} & & & & & & \\ & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}\dots & \color[RGB]{6,214,160}0 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}0.28 & \color[RGB]{6,214,160}0.68 & \color[RGB]{6,214,160}0.66 & \color[RGB]{6,214,160}0.71 & \color[RGB]{6,214,160}0.05 \\ \end{array} } \]

The Difference’s Bootstrap Sampling Distribution

difference.bootstrap.samples = array(dim=10000)
for(rr in 1:10000) {
    I = sample(1:n, n, replace=TRUE)
    Xstar = X[I]
    Ystar = Y[I]
    difference.bootstrap.samples[rr] = mean(Ystar[Xstar==1]) - mean(Ystar[Xstar==0])
}

  • Not bad. Our bootstrap sampling distribution looks a lot like our actual sampling distribution.
  • And, in particular, the intervals widths we calculate are very close: 0.156 vs. 0.163.
  • And that’s really all that matters. Q. Why?
  • Because that’s what we use to calibrate our interval estimate.
    • It’s our point estimate — the black dot
    • With arms wide enough to span 95% of the bootstrap sampling distribution — the red-shaded interval.

Why Does The Bootstrap Work?

Our Binomial-Specific Argument

  • Last class, we argued that we could estimate the sampling distribution of a proportion well enough.
  • Our argument used the parametric form of the sampling distribution: the binomial distribution.
    • The width of the binomial probability distribution doesn’t change quickly as the probability of heads changes.
    • So if we have an ok estimate of the probability of heads, i.e. the sample proportion \(\bar Y\) …
    • then we can use that instead of the actual probability of heads, i.e. the population proportion \(\bar y\).

\[ \bar Y \approx \bar y \implies \text{Binomial}(n,\bar Y) \approx \text{Binomial}(n,\bar y) \]

  • This told us the bootstrap worked because the bootstrap was doing exactly this.
  • That is, the bootstrap sampling distribution is Binomial with heads probability \(\bar Y\).
  • But the bootstrap still works. Why? Ultimately, it’s a subtle question with a lot of different answers.
  • I’m going to give you a simple one. One that’s very similar to the one we used last class.
  • Instead of equivalance to a parametric estimate, we’ll argue using approximation by one.

The Same Idea With Normal Approximation

  • Here I’ve drawn the same diagram as before, but with the binomial distribution replaced by a normal approximation to it. \[ f_{\mu,\sigma}(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}} \qfor \mu=p \qand \sigma=\sqrt{p(1-p)/n} \]

  • This is a good approximation. I can’t tell the difference between this diagram and the last one.

  • And we can repeat our argument using the normal distribution’s parametric form.

    • The width of the normal distribution doesn’t change too fast as its standard deviation \(\sigma\) does.
    • So if we have a good estimate of that standard deviation, e.g. \(\sqrt{p(1-p)/n}\) where \(p\) is the sample proportion \(\bar Y\) …
    • then we can use that instead of the actual standard deviation, which is the same where \(p\) is the population proportion \(\bar y\).

  • This argument generalizes well. It works, e.g., for the difference in proportions.

  • It works whenever the actual and boostrap sampling distributions …

    • are both approximately normal
    • and have similar standard deviations.

Approximate Normality for the Difference in Turnout

  • The actual sampling distribution of the difference in turnout is approximately normal.
  • So is the bootstrap sampling distribution. And they do have similar standard deviations.
  • Most things involving averages are approximately normal. This is a consequence of the Central Limit Theorem.
  • We won’t prove that, so you’ll have to take my word for it.
    • There’s a very nice proof that doesn’t really require any background you don’t have.
    • It uses an argument called Stein’s method that’s very interesting and very widely applicable.
    • But it’d take us a couple of lectures and we don’t have time for that.

The Same Idea for the Difference in Turnout

  • We can use almost the same picture to argue that the bootstrap works for the difference in turnout.
  • But there is a difference.
    • The spread of a proportion’s sampling distribution is determined by that proportion.
    • But the spread of the difference isn’t determined by the difference.
    • It’s determined by the two proportions we’re comparing.
  • This means we can’t summarize the range of interval estimates we tend to get as we did before.
  • i.e., by drawing a single interval around the 2.5th, 50th, and 97.5th percentiles of our sampling distribution.
  • Instead, we need to draw intervals at a range of widths.
  • Above, I’ve drawn intervals three intervals around each point estimate.
  • One each at the 2.5th, 50th, and 97.5th percentiles of the sampling distribution of interval estimate widths.

Summary

  • Most of the time, the bootstrap sampling distribution is a good approximation of the actual sampling distribution.
  • This means we have a general procedure we can use to calibrating interval estimates: we use the bootstrap.
  • The downside is that it’s a bit opaque.
    • We can use it to approximate our point estimate’s sampling distribution after we’ve run our experiment.
    • But that doesn’t tell us what it will look like before we’ve run it.
  • That matters when we’re designing experiments. It’s important to know what we will and won’t be able to learn.
    • e.g., it’s important to make our experiments big enough that we get the precision we need.
    • This is called power analysis and it’s a big part of experimental design.
  • Next class, we’ll talk about how to use normal approximation to do that.